It’s hard to walk by a card table full of puzzle pieces and not try to fit at least one into place. It’s just that desire for a mental challenge, and the way puzzles bring people together, that Fred Stuart relies on to keep customers coming back to his custom wood jigsaw puzzle business, ELMS Puzzles Inc., especially during the hectic holiday season.

A visit to the company’s factory, situated in the basement of his log cabin atop a hill in Harrison, evokes images of Santa’s workshop. Highly trained cutters saw each puzzle piece carefully, adding tricky shapes that make it difficult to discern whether the piece belongs on the edge or elsewhere. Buyers can pick a famous piece of art or submit their own images, whether it be an iconic scene of the Rockefeller Center ice rink and Christmas tree, a family photo overlaid with cutouts of the grandkids’ names for a 50th anniversary or even the more unusual, such as the gynecologist who wanted a picture of a uterus.

In the relaxing shop, where a handful of dogs roam, Stuart talks to Mainebiz about the business — to which people pay up to $3,250 for a one-of-a-kind, mahogany plywood puzzle — and a spin-off business where customers can rent a puzzle for much less. He’s a former engineer who took over ELMS when his wife and company founder, Betsy, died in 2007. An edited transcript follows.

Mainebiz: How did you get into the puzzle business?

Fred Stuart: After I met Betsy, she came to my house. I had a Hallmark cardboard puzzle on my dining room table. She put it together, boom, boom, boom. I found out that when she was 8 or 9 she had an Uncle Tuck who used to cut wooden puzzles, and he cut a “‘Betsy” into a puzzle. She never forgot it. In 1987, when we were married, she bought me a $99 jigsaw and we cut a puzzle. I looked at my hands and said, “Fred, you’re going to have nine fingers.”

She decided to form a company in 1987 called ELMS, her initials for Elizabeth Lee McShane Stuart. She was going whole hog from 1987 to 1990. In 1990, I sold my (engineering) company and semi-retired. We moved to Maine and Betsy designed this house with the basement as a factory.

MB: How did you grow the business in Harrison?

FS: Betsy started in 1990 with just three people. She had to train all the cutters, who were local people. Finally, in 1999, we outgrew the facility so we moved the cutting room into the garage. By then, we had six full-time employees. In September of 2000, right after we made this big expansion, Betsy was diagnosed with Stage III ovarian cancer. I began working the books for her. After she died, I stayed in that function full time. We were doing really good when Betsy was alive. She was an entrepreneur.

MB: Your clientele are affluent to afford your puzzles, so were you impacted by the recession?

FS: We were going along fine until Bear Stearns and AIG and all the stuff hit the fan in 2008 and cut our sales. We were doing reasonably well coming back from that until this government shutdown. It really affected business. It hasn’t come back yet. People who have money are sitting on it because they don’t know what’s going to happen next.

Right now, we’re running about 10 percent lower than this time last year. We gross about ($330,000) a year, and we’ve made a little money almost every year that we’ve been in business. I’ve been dumping money into the company to keep people working. In today’s times, selling expensive toys is not an easy thing to do. We have licenses with all the artists. Every time we sell one of our puzzles, they get a percentage of the sale. That’s a big expense.

MB: How is the rental business doing?

FS: We learned quickly that the custom business wasn’t going to keep us in business, so Betsy came up with a rental business. It’s a one-time fee of $75. They pay the cost of the rental, which is from $50 to $180, plus the freight. Now the rental business brings in more money than the custom business. So instead of paying $1,950 for a 600-piece puzzle, they can rent it for $115.