Swami Vivekananda's view
Swami Vivekananda's view that Jagannatha Temple of Puri is an old Buddhist Temple is already mentioned before. Dr. Bhau Lokhande has reviewed the subject in detail. Following is the summary of it. [Lokhande: 1979: 120] Lokhande summarizes
1. Poet Manohardas writes in his Amarkosha that Buddha is Arjuna and Krishna is param vishva atma. (Nagendra Nath Basu- Bhakti Margi Bouddha Dharma. Trans. By Narmadeshvara Chaturvedi, Allahabad, samt. 2018, p.173.
2. Buddha was also called Hari. (Ibid.,p.176)
3. Dr. Jadunath Sarkar has quoted a poem called 'Daru- Brahma' by a poet from Orissa, wherein Jagannatha himself says, I am incarnation of Buddha, I will give salvation to beings in Kaliyuga." (Sarkar, India Through Ages, p.177)
4. According to Nagendranath Basu, Buddhaswami of "Alekhlila" can be considered a Dhyani Buddha and Jagannatha as Bodhisattva or Padmapani and there is no doubt that the image of Lord Jagannatha was originally a Buddha image. (Ibid., p.177)
5. From a book Yashomatimalika it is clear that upto 41st regnal year of Mukundadeva, Lord Jagannatha was considered only as Buddha. That Mukundadeva was a great devotee of Buddha is known from Lama Taranath.
6. Behind Jagannath Temple at Puri, there is a huge stone image of Lord Buddha in Bhumisparsha mudra. In front of this image a big wall is erected. This image, which could have told many a thing to the students of Ancient Indian History, has now become a sealed book. This wall must have been erected so that the image of Buddha should not be visible to the people. The tradition of considering Lord Jagannatha as secret Buddha must have been started since then. Similar story is told about the image at Badrinath.
7. Ratha Yatra at Puri of Jagannatha, Balaram and Subhadra is nothing else, but a transformation of the Ratha Yatra of Buddha surrounded by Bodhisattvas and this Yatra was seen by Chinese pilgrim Fa Hein with his own eyes in the 5th century A.D (Sarkar, India Through Ages p.33)
Lord Jagannatha was worshipped by Tribals
Apart from account given by Dr. Bhau Lokhande there are other points to be noted about Jagannatha of Puri.
There is a legend that Lord Jagannatha was a deity worshiped by a tribal chief, secretly in a cave. Later this deity was to be interned in a wooden image. There images are buried and new images installed from time to time, but Navi-Padma of Shri Jagannatha is transferred from the old to the new images. It is believed by historians that it contains the tooth of the Buddha. The details of legend told to the tourists are as follows:
The first temple, according to the legend, has been built by King Indradyumna. He felt that there was a concrete presence of the Lord somewhere near about, but he did not know where. He sent four seekers in four directions. One of them, Vidyapati, a young Brahmin, reached a forest and stayed as the guest of tribal chief, Visvavasu. The chief's charming daughter, Lalita, fell in love with him and they got married.
While there, Vidyapati came to know that Visvavasu worshipped some secret deity in a cave. Through Lalita he traced the cave and at once realized that this was the Divine, and escaped to Puri.
Later King Indradyumna apologized to Visvavasu, and obtained his consent for installing the deity in the temple he had built. But the deity was to be interned in a new image, to be carved out of a log that had come floating in the sea. A mysterious old man offered to carve the image. As the image resembled the King's vision, he was allowed to proceed with his work, on the condition that nobody would open the closed room in which he would work before a certain time. But after some days, Gundicha, Indradyumna's queen grew impatient. She pushed open the door. The craftsman disappeared, leaving the three images incomplete in which form they are seen to this day. The images are buried and new images installed at the interval of many years, when the astrological calculations demand it. But the Navi Padma (the lotus shaped navel) of Shri Jagannatha is transferred from the old to the new image. What does this Navi Padma contain? Some historians believe that it contains the tooth of Buddha.
Evidence of a Text from Sri Lanka
Sri H. L. Kosare has mentioned:
"A Simhalee Buddhist Text from Sri Lanka, Datha vamsho, is considered to belong to fourth century A.D. It says 'The ruler of Kalinga was one Guha-shiva. He was the vassal of the Emperor of whole of Bharat and Jambudwipa ruling from Pataliputra. This emperor was the worshiper of Brahmin dharma and arya dharma.' From this it appears that, this reference is of Samudragupta and this king Guha or Guhak was his vassal. The text further mentions that, the Emperor at Pataliputra was complained about this vassal was worshiping the 'dead bones' and abuses the arya devatas. (datha vamsho, J.P.T.S. p. 167, verse 72-94). This proves that Buddha's Tooth was being worshipped in Kalinga state, about which a complaint was lodged with Samudragupta." [Kosare H. L.: 1989: 245]
Caste barriers are weak at Jagannatha
It is well known that Buddhists never believed in caste, and those castes, who in Hindu system of Varnas are considered to be very low, are also treated on equal terms by the Buddhists. It is therefore clear, that those places of worship where respect and honour given to low caste people is comparatively better, should be considered as old Buddhist shrines. Such is the case of Temple of Lord Jagannatha of Puri. Catholicity of Lord Jagannatha about caste is well known.
Prof. Ghurey observes:
"The officiating priest of the famous Temple of Jagannatha is a barber, food cooked for the deity by him being acceptable to all but the most orthodox amongst Brahmins..." [Ghurye G. S.: 1969: 27]
J.H. Dave observes:
"Jagannatha is a god of the people. High and low are enjoined to eat here together. In presence of the Lord, all are equal as all barriers of caste, race and faith are transcended. To deny the mahaprasada of the Lord, which is always bloodless, is said to invite the wrath of the despised God." [Dave: 1970: 45]
Dave further observes:
"The temple attendants are divided into 36 orders and 97 classes. The leading one is the Raja of Khurda who calls himself by the lowly title of "sweeper to Jagannatha"..." [Dave: 1970: 46]
Puri is Dantapura
Dave observes:
"There are 24 high festivals during the course of the year. the greatest of them all is, of course, the Car Festival or the Ratha Yatra. Fa Hien's description (5th century A.D.) of the yearly procession of the Buddha's sacred Tooth applies greatly to this procession." [Dave: 1970: 46]
He further confirms that Puri is the same place as Danta Pura. This is what he says.
"Following on the early Hindu period of the worship at Puri, according to Dr. Rajendralal Mitra, there was a Buddhist period which in turn was followed by the period of Krishna or Vishnu worship. The peculiarities of Jagannath worship, its catholicity, its broad basis, its ignoring of caste barriers, and the similarity of the Ratha Yatra or Car Festival with the procession of Buddha's tooth - these and similar other factors tend to support the view that Puri was the same Danatapura where the sacred relic of Buddha's tooth was also situated and preserved. It was taken out every year in a procession with great pomp and devotion and subsequently removed to Ceylon." [Dave: 1970: 43]
Views of Prof. Rao
Lastly, it would be desirable to see what Prof. T.A. Gopinatha Rao has observed a long time back. "...The temple of Jagannatha is believed to have belonged to the Buddhist at one time and to have been converted into a Vishnu Temple at a later date. The image of Jagannatha is an ill-shaped log of wood with two big eyes marked on it rather prominently. Once in twelve years the log is renewed, the log being brought mysteriously from some unknown land. This is utilized for carving a new image of Jagannatha, in which some ancient relic is considered to be embedded. It is the insertion of this relic which sanctifies the new images. This relic is believed by some to be relic of Buddha..." [Gopinath Rao: 1985: 273]
For those who always wondered as to why the images are ill shaped, this explanation should be enough, as it is not the wooden image but the relic, that is of importance.