"You're doing it wrong!"
If I had a dollar for every time I've heard an overbearing Scrum Master, misguided "Agile expert" or overzealous individual claiming to be an Agile coach say these or similar words, I'd be a rich man indeed.
What is right or wrong is often situational in Agile. The Scrum Master who insists that you must sit with the team may or may not be correct as to where you would be most effective. The idea that there is one and only one correct answer to the question of where you should sit is too simplistic. And the Scrum Master who dictates an answer somehow missed the essense of the Scrum Master role when they attended their Scrum training. The optimal place to sit is discovered through practice and the choice of where to sit is yours (see self-organization at http://agilemanifesto.org/principles.html).
The simple idea that the customer or "business expert" should sit with their Agile team is not so simple in execution. For starters, what exactly does "sit" mean? Does sit mean that you must be physically in the room with the team or is "sitting with the team" more a question of accessibility and reaction time? What if your business expertise is made more effective with regular face-to-face interactions with your customers or business peers in the field? What is the proper balance of in the team room time versus in the field time? The correct answer is, of course, "it depends", and what it depends on is what you experience through practice. And, the proper balance is volatile. What works today might not work tomorrow. It is not so much finding balance as it is seeking balance.
Here are a two things to keep in mind as you seek a balance that works for you:
Does your team get the answers to their business questions fast enough? Delays in getting clarity or feedback from you on their business questions means that your team is likely wasting time, either by delaying working on the most important needs, or perhaps worse, making incorrect assumptions that will require future work to correct. If your team is experiencing unacceptable delays or creating unacceptable future work, you might want to spend more time with them. (see 80/5 guideline)
Are there too many "late" surprises? You should never be surprised by the team's product during a product review at the end of an iteration. If there are missed opportunities to catch a surprise early and take corrective action sooner and probably cheaper, you might want to spend more time with your team.
Where you sit does matter to the effectiveness of your team in meeting your shared goals. Pay attention to your team's business informational needs and they will pay attention to building what meets your business goals.