A wooden cabin by a lake in the middle of rural Ireland. A freezing midwinter morning. Four musicians are in the living room. A bass guitar wobbles between two alternating notes. A complex drum fill crashes for half a bar, then stops. Two atonal synthesiser stabs, each from a different keyboard. From somewhere else in the cabin, a man's voice moans: "Again." The musicians go through the cycle again: bass wobble, drum fill, synth, synth. "Again," the voice wails, from a different room. Bass, drum, synth, synth.

I'm watching the band play on a monitor from the room next door, huddled with other technicians out of shot. The man shuffles past us and shouts through the wall: "Again!" Bass, drum, synth, synth. He is wearing a giant fibreglass head with big blue eyes and a severe parting painted on to it. The technicians are trying to keep their faces straight. The man opens the door and pokes his head into shot: "Again!" Underneath the head is Michael Fassbender, though at this stage, it can't be proved.

We are on set of the movie Frank, an eccentric comedy about an eccentric rock band led by a man who has wandered off the spectrum of eccentricity and into mental illness. In a movie market crowded with prestige biopics, fictional comedies, concert movies, documentaries and mockumentaries, Frank is aiming for something a bit alternative. A lot more alternative, actually, as this morning's scene suggests. The band in the movie goes by the "so edgy no one can even pronounce it" name of the Soronprfbs. They have come to this muddy corner of County Wicklow to record their new album – spurred on by their obsessive, inscrutable, fibreglass-headed leader. They have been here a week in real life and 18 months in the movie, which explains why Domhnall Gleeson, the keyboard player, is sporting a long straggly beard.

Indie aficionados of a certain vintage will recognise the head as a homage to Frank Sidebottom, alter-ego of Mancunian musician Chris Sievey, who died in 2010. Sidebottom was a genuine British eccentric, part-punk provocateur, part-end-of-the-pier parody. His gigs were unpredictable, shambolic affairs, often involving audience participation, puppets, references to Sievey's home village of Timperley, and the mauling of popular hits such as I Should Be So Lucky and Anarchy in the UK. He sang along to a lo-fi synthesiser in a cheery nasal whine, was fond of using the word "bobbins", and tended to end every song with the phrase: "You know it is, it really is." It is hard to imagine an act less geared towards the big time, which is exactly what his fans liked about him.

Understandably perhaps, certain adjustments were necessary for Frank, the movie. It was co-written by Jon Ronson, inspired by his time in Sievey's band in the late 80s, but this Frank is American and present-day, explains director Lenny Abrahamson from beneath a woolly cap and several layers of outerwear.

"We've moved so far from Jon's original memoir that began the process," he says, over a rapidly cooling lunch on the catering bus. "What you've got left is a kind of debt to Frank Sidebottom in the aspects of the sensibility, and the head. I hope it's a tribute to him, but it's also inspired by lots of other outsider musicians and strange, creative but ultimately damaged individuals. So we looked at Captain Beefheart and Frank Zappa and Daniel Johnston, lots of people like that." Beefheart notoriously held his band under virtual house arrest in California in the 60s to make his magnum opus Trout Mask Replica – but at least they were somewhere warm.

There is a balance to strike here, or is it a contradiction? Like a sprawling desert, the "mainstream" inevitably subsumes all things "alternative" over time, in both independent music and cinema. Cult musicians such as Sidebottom, Beefheart and Johnston are guarded by their fans like endangered species in a nature reserve. By opening up these backwaters, there is a danger Frank could earn the enmity of the very pop connoisseurs it hopes to attract. In fact, Frank's story addresses exactly this dilemma: between artistic integrity (and obscurity) and commercial success. The band are dragged toward the latter by Gleeson's ambitious new keyboard player, Jon. This Twitter-savvy newcomer secures the band a gig at SXSW, the Texan music festival where the underground traditionally goes over. It turns out to be their undoing.

"Everybody else in the band is madder than Jon," explains Gleeson between takes, in the character's middle-class English accent (if he slips into his native Irish between takes, his vowels start going, he explains), "but underneath the politeness, he has ambition. He puts pressure on the band to be more mainstream." He's smoking a cigarette through a holder so as not to ruin his straggly beard.

You couldn't accuse Frank of "selling out", exactly. It's not a big budget film, and there is an aura of playful experimentation that seems to be sustaining the cast through the mud-spattered adversity. Abrahamson, an emerging force in Irish cinema, has already proved his outsider credentials. His debut, Adam & Paul, followed two comically hapless low-lifes around the more squalid areas of Dublin, while 2007's Garage centred on a simple-minded petrol station worker. Frank is less of a departure for him than it looks, he says, especially in terms of the tone, which he describes as "a celebration of a certain kind of delicate slapstick, which I think has massive potential to express emotion. Because the nature of this band is to muddle through chaotically, it means we don't have to be po-faced about it. We're not making Control here."