Mrs Rose Diamond, a small, sprightly, smiley 92 year-old, asks if I’d like some corn on the cob. Her son, sitting next to her, wonders if I’m enjoying my hot dogs. Outside, the bottom of the garden is being pounded by the Malibu surf.

This is an average Sunday afternoon at Neil Diamond’s weekend retreat in one of California’s loveliest neighbourhoods. Once past the imposing steel gate, the open-plan house isn’t especially big or extravagant, but the frontage is spectacular. His beloved mother visits every Sunday, having moved from the east coast to be near her son (his father died in 1985).

Related Articles

• Neil Diamond, BBC Electric Proms

Even if I hadn’t grown up on his music – courtesy of my mother’s boundless enthusiasm for Red, Red Wine, Cracklin’ Rosie, Sweet Caroline, I Am… I Said, Song Sung Blue, Hello Again – this would be quite a thrill. This man has sold 128 million albums, making him the third biggest recording artist of all time.

Forgive the momentary abandonment of cool dispassion, but Neil Diamond is serving me lunch.

‘The last time I did an interview in my home it was with Rolling Stone and it was in 1976,’ he reflects as the smell of cooking drifts from the kitchen.

‘It turned out to be a disaster. My wife at the time was gonna cook up some steaks for us all, but my dog got on the table and stole all the steaks,’ he grins as his spaniel, a rescue dog named Poker, scampers around his shins. ‘I had to go out for Kentucky Fried Chicken. Let’s hope this one works out better.’

Diamond has such a quiet, sober and unassuming manner that it’s sometimes easy to forget his achievements. He was the writer of the biggest-selling song in the United States in 1966 (the Monkees’ cover of I’m A Believer). He is half-responsible, alongside old high school contemporary Barbra Streisand, for one of the greatest duets of all time (1978’s You Don’t Bring Me Flowers).

In 1980, he was the highest paid debut actor ever ($3.5 million for The Jazz Singer). He was almost cast, before Robert de Niro, as Travis Bickle. ‘I knew this producer, and we were having dinner and he said: “You know, I think you might be the right guy for this picture I’m doing. It’s called Taxi Driver.” I guess he saw me as strong, silent and crazy.’

He was the dance partner of choice for Princess Diana (at a Reagan-era White House ball in 1985). More recently, he was the undisputed hit of Glastonbury 2008 (festival organiser Michael Eavis in the run-up said: ‘Normally he charges around £750,000. I can’t tell you what we’re paying him, but it’s very little money. He just wants to be here, bless him’). And last week he was the closing star of this year’s BBC Electric Proms concerts (held last night at the Roundhouse in Camden, London, and broadcast on BBC Two on Saturday, November 13).

In his late sixties Diamond enjoyed a career rebirth via two beautiful, startlingly stripped-back albums he made with Rick Rubin, 12 Songs (2005) and Home Before Dark (2008). The producer had worked similar magic with Johnny Cash at the twilight of his career. And even as he approaches his 70th birthday in January, Diamond is already thinking about his next album, a set of songs written with different collaborators.

I’m not entirely convinced by his enthusiasm for working with Fred Durst, late of goon-rockers Limp Bizkit. But in his hands, with his track record, I suppose magic can come in myriad forms.

Diamond is the rare artist covered by Elvis Presley, Frank Sinatra and Puck, star of hit television show Glee (all have performed Sweet Caroline). This month he, in a way, returns the favour, by releasing his own album of cover versions.Dreams, Diamond’s 32nd studio album, features his caramel-baritone take on, among others, the Beatles’ Yesterday and Blackbird, Bill Withers’ Ain’t No Sunshine, Elton John’s Love Song and Leonard Cohen’s Hallelujah.

‘As opposed to a writing album, this is a singing album. The main thing was that I had to feel like I could sing it and do it differently than the commonly known version,’ Diamond explains of the reasons behind his song choices.

‘I didn’t want to be copying or covering that same record. I wanted to have my own take on it. And I think we did pretty good, just because I’m different and I saw those songs a little differently.’

In 1961, Neil Diamond was a Jewish kid from Brooklyn at a crossroads. At his shopkeeper parents’ urging, he was studying medicine at New York University. He was a keen amateur fencer, too, but his heart lay with music. Diamond had already had some success with the first song he wrote, Hear Them Bells. He wrote it in an attempt to impress a girl from his neighbourhood.

‘As primitive and clichéd as the song was, she was evidently moved because we started dating and we went steady and then we got engaged and then we got married and then we had children.

‘Then unfortunately we got divorced five or six years later. I should have realised then the potential power of songs,’ he chuckles, ‘and been a little more wary.’

Diamond opted to drop out of his studies and began to make a living as a writer on New York’s Tin Pan Alley, the behind-the-scenes engine room of popular song, while hoovering up as much music as he could. He had learnt to play guitar by listening to folk group the Weavers.

The exciting new form of rock ’n’ roll had pricked up his ears. Meanwhile, a song performed by the Everly Brothers – one of their LPs was the first record he bought – had been all over the radio.

‘I loved Let It Be Me,’ Diamond says of Don and Phil’s 1960 hit. ‘I was a huge Everly Brothers fan. The harmonies were just spectacular. And that was a particularly beautiful, very melodic song.’

He discovered that the song was originally a French composition, written by Gilbert Bécaud. He and the Frenchman would go on to write a handful of songs together, including Love on the Rocks and September Morn, both from the soundtrack of The Jazz Singer.

Fifty years on, Neil Diamond sings Let It Be Me on Dreams. As well as the covers of other artists’ songs, there’s a reclaimed, reimagined version of his own I’m A Believer. Five years after he swapped medicine for music, that song, covered by the Monkees, sold a million copies in 48 hours.

‘It’s my own take on it,’ says Diamond of his new, slowed-down, warm and stately version of one of his canonical songs. ‘After all these years, this is how I do it now. This is the way I feel it now.’

Diamond is a symbol of American culture and he was the most profitable touring act of the Nineties, earning $182m. Today, this tall, lean figure talks in a calm, methodical manner. But as he demonstrated at Glastonbury, stick ‘the Jewish Elvis’ on a stage and there are few showmen who can touch him.

I’m A Believer was the song that made Diamond’s name. But he’d already endured five years of rejection and failure on Tin Pan Alley. He had been a young writer in the famed Brill Building, desperate to emulate the success of those stellar talents ahead of him: Burt Bacharach, Carole King, Lieber & Stoller.

‘They were already established. They were recognised as great writers and they were. Burt had written for and produced Dionne Warwick, Lieber and Stoller were like gods, and Carole King was a prodigy. I was just a guy knocking around on the streets trying to get some of my dumb songs heard.’

Neil Sedaka, too, was another writer Diamond looked up to – not least because they had attended the same school. ‘He’s a year or two older than I am and was very famous in high school. So I watched him very carefully. I was trying to be a professional writer at that point and here was somebody who had made it – and he was from my neighbourhood.’

Diamond never encountered the Beatles personally. But he did witness their impact on Tin Pan Alley.

‘I remember I was still in the Brill Building when they arrived in the US. All the staff writers gathered around and listened to this new group from England that was all the rave and all the talk.

‘I thought: “OK, they’re pretty good.” We thought maybe it was just like a teenage sensation because the kids were going crazy over the Beatles. But they did change the way the music business was done in the United States, no question about it.’

The shock waves of the Fab Four’s invasion of the US, he says, were profound. ‘First of all, the writing of Lennon and McCartney signalled the emancipation of the songwriter, who had always been the low man on the totem pole. Paid the least. Least respected. Least recognised. But suddenly the songwriter was on a par with the big stars – these guys could not only write but they could sing as well.’

So began the end of the Tin Pan Alley era. ‘People realised it wasn’t necessary to have somebody own your copyright to your songs and bring them around to try to get other people to record them,’ says Diamond, who has hung on to the copyright of all his songs since.

Buoyed by the success of I’m A Believer, Diamond made the transition from backroom writer to front-stage performer. One of his earliest tours was with Herman’s Hermits and the Who. In his youthful naivety he thought he would show the latter, then on their first tour of the US, a thing or two.

‘I had no idea what the Who was doing and I was pretty full of myself as far as performing goes. They’re just a group and how could they possibly follow me? Then they started to destroy the stage – yeah, they did a pretty good job of following me,’ he says wryly.

On Dreams, Diamond covers Midnight Train To Georgia, a 1973 hit for Gladys Knight and the Pips. The lyric is about a musician who moves to Los Angeles to follow his dreams, but things don’t work out. Diamond made the same move in 1969, but was an instant success. He had a home – complete with mirrored ceilings – in Laurel Canyon, the heart of the LA rock scene then.

‘I never went out,’ says the man whose first hit under his own name, Solitary Man, remains something of a signature song. ‘I stayed home and I wrote my songs. And I went on the road. I didn’t hang out with anybody. I know Joni Mitchell was there and a couple of guys from Crosby, Stills and Nash. There was a whole group of writers, talented people, who were in Laurel Canyon.’

They were also, famously, wrapped in the boozy, narcotic excesses of that period. Why not Diamond?

‘I was a father,’ he shrugs. He has two daughters with his first wife and two sons with his second. ‘And I saw myself as a father. And a father doesn’t go out and party every night, and get involved in drugs. I’ve always been afraid of drugs. So that just never was my thing. I was all about the music, the performances and my kids. I didn’t want to know about anything else.’

He took four years off to be with his family – ‘my first marriage broke up [in 1969] and I just didn’t want it to happen again. I’d been out touring for over six years straight. I decided to concentrate on giving my family the time they deserved’ – yet Diamond remained one of the best-paid performers of the Seventies.

He broke box-office records all over the US and in 1972 signed a hugely profitable contract with Columbia. Last year, he renewed that deal with a new five-year contract – as he tweeted at the time, that’ll make 43 years with the same label. None the less, he remained a man – and an artist – apart on the LA scene. Even now his Twitter account shows that, while he has 165,412 followers, Diamond ‘follows’ precisely no one.

On Dreams he sings Gilbert O’Sullivan’s 1972 hit Alone Again (Naturally) and the Eagles’ 1973 smash Desperado. The lyrics of both seem to fit his nature. Ask him if he moved in the same circles as the Eagles – both acts are now managed by Irving Azoff, perhaps the biggest manager in the business – and he replies smoothly: ‘Well, actually, they moved in the same circles as me.

‘When I moved out here I was a hit artist. When I was playing [famous LA club] The Troubador they were in the bar, outside, trying to figure out a way to get a record on the charts. [Eagles singer/writer] Glenn Frey told me that! So,’ he chuckles, ‘I was inside singing my hits and they were outside trying to get a hit!

‘But I’ve never really moved in anybody’s circles for some reason,’ he adds.

Even his legendary collaboration with Barbra Streisand came about by accident rather than by design. ‘Barbra and I met very casually,’ he remembers of he and Streisand’s hookup on the immortal You Don’t Bring Me Flowers, ‘and it really was an accident more than anything else.’

Diamond had written a 60-second song as the themetune to a sitcom called All That Glitters. ‘The roles of men and women were reversed: the men stayed home and took care of the babies, the women went off to work.

‘When I was told about it by the producer, I proposed doing a torch song, for a man. And they’re usually sung by a woman: you hurt me and you didn’t do this and you broke my heart. So that’s why a man is singing “you don’t bring me flowers”.

‘But they turned the song down and the series didn’t make it anyway – it did only a few episodes.’

But Diamond liked the song enough to expand it and record it for one of his own albums. Streisand heard it, loved it and recorded her own version. An enterprising radio DJ in Kentucky heard both versions and edited them together. Suitably impressed, Diamond and Streisand decided to go into the studio and record a ‘proper duet’.

Despite his success, and his name (it’s his real name; mercifully he decided to forget an early idea to adopt the stage name Ice Cherry), not to mention his extravagant Seventies shirts, Neil Diamond has never been one for glitz. He met Frank Sinatra a few times and talks approvingly of his ordinariness in ‘real life’.

‘Frank was always very nice and very likeable. I wouldn’t say he was charismatic. Charisma comes into play when you’re doing your thing. When he was in front of a microphone he was definitely charismatic.’

Elvis Presley, meanwhile, was a bigger, rowdier force. They met in Las Vegas, when Diamond was on holiday. ‘I wanted to go see Elvis and enjoy the show. He was gracious enough to introduce me from the stage. I got up and waved to everybody, and the audience started chanting: “Get up and sing!”

‘I really didn’t want to interrupt his show. And he knew it, too. He said: “No, Neil’s on holiday, he’s here to enjoy the show, we’re gonna let him do that.” It was a nice moment, a flattering moment.’

But Diamond learnt from Presley. After the latter’s death in 1977, Diamond observed: ‘I learnt from Elvis to avoid becoming public property. Nothing is worth that.’

Now he keeps himself to himself, here in Malibu, or at his main residence in Beverly Hills, his New York apartment or his cabin in the Colorado mountains. Or at his studio, Arch Angel, in central Los Angeles, which he’s owned since the mid-Seventies and seems to visit almost daily.

Neil Diamond has always been Mr Mellow, Mr Low-Key, Mr Agreeable. In the early Nineties, when he and his second wife split after almost 30 years’ marriage, they sorted things out themselves over the phone, splitting his fortune down the middle. At a reported £75m, it was the second-most expensive ‘celebrity’ divorce ever, according to Forbes magazine.

‘She earned every penny,’ Diamond says. She had helped him keep ‘my sanity intact because she’s saner than I am. None of my career would have been the same without her.’

He’s currently single, his relationship with Rachel Farley (an Australian 30 years his junior whom he met while on tour Down Under in 1995) having ended two years ago. Is Diamond content being alone again, naturally or otherwise?

‘Well, I don’t know if I’m content,’ says this grandfather of six with typical reflective candour. ‘I like having a woman. I like having someone to come home to, to make all of the hard work feel worth it. And that someone is sharing it with me, the joy and the downs of it, the hard part of it.

‘So, right now, I’m content, but I don’t think I’ll be content for too long. I need someone with me. And I want someone.’ So, the singles columns? He laughs at the suggestion.

‘Well, you know, I find that these things just tend to happen. It’s like writing a great song. You can’t plan to write a great song. It just happens to you. It drops in your lap. It’s the same thing with a woman.’