Gessel, Ira M.; Xin, Guoce The generating function of ternary trees and continued fractions. (English) Zbl 1098.05006 Electron. J. Comb. 13, No. 1, Research paper R53, 48 p. (2006). Summary: Michael Somos conjectured a relation between Hankel determinants whose entries \({1\over 2n+1}{3n\choose n}\) count ternary trees and the number of certain plane partitions and alternating sign matrices. Tamm evaluated these determinants by showing that the generating function for these entries has a continued fraction that is a special case of Gauss’s continued fraction for a quotient of hypergeometric series. We give a systematic application of the continued fraction method to a number of similar Hankel determinants. We also describe a simple method for transforming determinants using the generating function for their entries. In this way we transform Somos’s Hankel determinants to known determinants, and we obtain, up to a power of \(3\), a Hankel determinant for the number of alternating sign matrices. We obtain a combinatorial proof, in terms of nonintersecting paths, of determinant identities involving the number of ternary trees and more general determinant identities involving the number of \(r\)-ary trees. Cited in 1 ReviewCited in 27 Documents MSC: 05A15 Exact enumeration problems, generating functions 05A10 Factorials, binomial coefficients, combinatorial functions 05A17 Combinatorial aspects of partitions of integers 30B70 Continued fractions; complex-analytic aspects 33C05 Classical hypergeometric functions, \({}_2F_1\) Keywords:Hankel determinants; plane partitions; alternating sign matrices; generating function; continued fraction; hypergeometric series; determinant identities PDFBibTeX XMLCite \textit{I. M. Gessel} and \textit{G. Xin}, Electron. J. Comb. 13, No. 1, Research paper R53, 48 p. (2006; Zbl 1098.05006) Full Text: arXiv EuDML EMIS Online Encyclopedia of Integer Sequences: a(n) = 2*(3*n)! / ((2*n+1)!*(n+1)!). a(n) = binomial(3*n,n)/(2*n+1) (enumerates ternary trees and also noncrossing trees). Number of alternating sign 2n+1 X 2n+1 matrices symmetric about the vertical axis (VSASM’s); also 2n X 2n off-diagonally symmetric alternating sign matrices (OSASM’s). Number of alternating sign 2n+1 X 2n+1 matrices symmetric with respect to both horizontal and vertical axes (VHSASM’s). a(n) = binomial(3*n+1,n)/(n+1). a(n) = 2*det(M(n; -1))/det(M(n; 0)), where M(n; m) is the n X n matrix with (i,j)-th element equal to 1/binomial(n + i + j + m, n). 3rd-order Patalan numbers (generalization of Catalan numbers). Number of cyclically symmetric transpose complement plane partitions in a 2n X 2n X 2n box. Expansion of F(1/3,2/3;1/2;27*x/2) / F(1/3,-1/3;-1/2;27*x/2).