As you read this, I will be somewhere warm and windy in the Caribbean. Just me, with my windsurf gear, skimming across the gloriously blue water – with my smartwatch firmly strapped to my wrist tracking my every movement. It’s one of the few times my smartwatch is pushed to be functionally useful. The GPS features map my distance, location, and speed, while the heart rate monitor and other gadgets track the inner-me: my heart rate, breathing, calorie burn, and so on. A complex fusion of components come together to track almost everything about me, but of course, all I really care about is my speed.

And that’s the crux of the challenge for smartwatches. The strong emphasis that the manufacturers place on the health and fitness functionality is of questionable value to us mere mortals. Sure, we all think we care about details like how our workouts went, how many calories we burned, and even how well we sleep. But does it matter? I’ve argued in the past that sleep tracking is fairly useless but let’s expand that premise to the rest of our workout activities.

When I’m not windsurfing, I’ve been known to glance in the general direction of a gym. My smartwatch faithfully tracks it all: the weights I lift, the distance I run, and – more commonly – notifying me of its disappointment when I miss a workout (or three). It’s like the disapproving look of a spouse when you forget to do the dishes … again.

But the main reason I value my watch is because I use it to listen to music while I work out, so I can leave my phone at home. For all the fancy features that manufacturers keep cramming into this small device, the only fitness related one that really matters, at least to me, is playing music. Even step counting is of minor importance, unless I commit to forcing out a few additional steps to meet my goals. Unlikely. So why do I keep my smartwatch charged and firmly on my wrist when I’m not windsurfing, if not for the sake of fitness tracking? Simply put, the two killer apps for me are the most basic: it tells me the time and informs me of notifications (texts, calls, and so on) so I can leave my phone on mute in my pocket.

While many people get – or think they get – more value out of the fitness features of smartwatches, manufacturers need to evolve beyond these features soon. According to the latest Connected Intelligence Wearables Ownership Report, 38% of U.S. adults own a smartwatch but the market has flattened, with no significant increase in ownership in the past six months, despite the holiday shopping season. And I would suggest that a reason for the flattening is that many people who care about tracking their fitness already own one. To grow the market, manufacturers will need to build out more non-fitness use cases (beyond my basic uses of time telling and receiving notifications). Currently, third party apps that provide access to weather, news, and music/podcasts see the highest everyday use among smartwatch owners. This non-fitness app use is increasingly important to keeping the existing owner committed during times of less activity, as well as persuading the remaining 62% of U.S. adults that a smartwatch is worth owning.

Of course, this is dangerous territory. An expansion like this potentially changes the role of the smartwatch from being a mobile phone accessory (albeit a quite sophisticated one) to being an independent device that can reduce our reliance on our phones. Does such a move mean we no longer need a smartphone? Probably not. But does it mean that the phone becomes less important to us, with less need to replace it quite so often? Quite possibly, especially when used in conjunction with earbuds. But, since smartwatches are made by manufacturers who also have a vested interest in people buying phones, that’s a tricky move and one that may be resisted. After all, who wants to compete against themselves, and put an existing revenue source at risk? However, if these same manufacturers do not seriously look at this opportunity, then it is only a matter of time before someone without any phone baggage in their portfolio does. And at that point, it will be too late for the smartphone-smartwatch OEMs to react.

But before any of that happens, I’ll continue to use my watch for the basics of simple buzzing notifications and, of course, tracking my windsurfing speed. Speaking of which, I’m headed back off to the water. But don’t feel too bad. I’ll be back in New York with a rapidly fading tan before you know it, with only my app’s speed data to show I was even here.

Get insights straight to your inbox