Diss is a lovely town, and so is Harleston - but the area in between is sheer hell for church explorers. Having experienced the fatuous churchwarden at Scole, I was already not in a very good frame of mind - being treated with suspicion by people who call themselves Christians always makes my heart sink - and as I headed on through Billingford, Thorpe Abbots and Brockdish, all locked without keyholders, I was descending into something approaching despair.

What a delight to reach Harleston then! It was market day in this tiniest of East Anglia's market towns, and people were busying about. The pubs and shops were all gaily bedecked, and not a chainstore in sight. Just off the market square on the road into Suffolk sits St John the Baptist. And bless it, it was open.

The church was built in the 1870s to replace a ruinous medieval church on the market place. That had been built as a chapel of ease to mighty Redenhall, and the modern church still has the feel of a chapel, built by diocesan architect Richard Phipson, most famous for the interior of Norwich St Peter Mancroft and the complete rebuilding of Ipswich St Mary le Tower. He did a lot of work in the Waveney valley, but as far as I know this is his only complete church outside of Suffolk.

It is built in a Decorated style with a long clerestory and low aisles. Phipson is noted for observing the letter of architecture rather than the spirit, and this can produce masterpieces like St Mary le Tower. But many of his restorations are middle-brow, some of them dull in the extreme. He seems to have gone through some sort of psychedelic drugs period in later life, resulting in the bizarre spires at Great Finborough and Woolpit in Suffolk, but that was all in the future when he rolled up his sleeves at Harleston.

A large vestry on the south side creates a sense of the cruciform, and the eastern apse, an unusual feature for Phipson, creates something of the impression of a small French cathedral. The west entrance is glorious, but the regular knapped flint, a mid-Victorian passion, is not terribly attractive, reminding us somewhat of its later customary use for cemetery chapels and crematoria. And so we step inside, through the south door rather than the west.

At once, the sense of being in a French church is magnified. The western bays of both aisles were boxed off in the 1980s, that on the north side as a bookshop and coffee area, on the south as a narthex; you step into this outer church as you would in Boulogne or Bergerac. And then, to complete the illusion, into the body of the church, and all is dark and high, flanked by lugubrious glass, the only light coming from the apse at the east end - surely we must be in France.

In France, of course, the east end would have been reordered in a democratic way as a result of Vatican II; here, as in much of the Anglican world, there is still the medieval heirarchy. The glass is very interesting, because the whole church is a single commission from the O'Connors in the 1870s. That in the north aisle has since been removed and replaced in the 1980s with flowing images something in the style of Surinder Warboys. The panels in the apse tell the story of Christ from Annunciation to Resurrection - my two favourite panels are the deposition and the three Marys at the tomb. Scenes from parables and the life of Christ fill the nave. The best is undoubtedly the lame man being lowered through the roof of the building where Christ is teaching - he wears Victorian workman's clothes. The raising of Lazarus is also good. The Good Samaritan is the odd one out, because instead of the three lights forming a single subject it tells the story in three images.

The modern glass depicts the river Waveney and its wildlife, angel musicians and a nimbused figure beneath an autumnal tree, who I assume is St John the Baptist. The O'Connors' 1870s sequence of scenes in the life of St John the Baptist in the west window should also not be overlooked.

This is a lovely church, not wholly in the modern fashion by being so dark and shadowy. What it really needs, of course, is what it would have in France; statues of Saints and shrine altars in the aisles, with old ladies and earnest young men wandering in off the market place, lighting candles and saying private devotions. It may well be that Phipson had them in mind, because he designed his churches for the kind of worship he enjoyed; ritualistic, incense led, with choral voices echoing around the high nave. He saw no great difference between his own Anglo-catholicism and the traditional Catholicism of the continent. I'm so glad nobody has wrecked this place by taking out all the glass and turning it into an evangelical preaching barn. What's left to us makes the rest of it easier to imagine.

Simon Knott, June 2005