More than 1460 years ago in the Liang dynasty, Tanyu, the master priest, under orders of Emperor Liangwu, built Baozhuangyan Temple to store the Buddhist bones brought from Kampuchea. During the Northern Song dynasty, a great writer - Su Shi, on a visit to this temple was attracted by the six banyan trees surrounding it and wrote the inscription "Liu Rong" (Six Banyan Trees). This temple has since been known as Temple of the Six Banyan Trees.

The temple was burned down and rebuilt in the Northern Song dynasty (960-1127).

Flowery Pagoda, built in 1097, is the major structure in this temple. Named for its Colorful exterior, the pagoda stands 57 meters high in the center of the temple, having a bronze column with 1,000 Buddhist sculptures on its top. Viewed from outside, the pagoda has nine stodes while in fact it has 17 stories inside.