To render an object by ray tracing, one or more rays are shot from the viewpoint through every pixel of the image plane. For reflective and refractive objects, especially for multiple levels of reflections and/or refractions, this requires many expensive intersection calculations. This paper presents a new method for accelerating ray tracing of reflective and refractive objects by substituting accurate-but-slow intersection calculations with approximate-but-fast interpolation computations. Our approach is based on modeling the reflective/refractive object as a function that maps input rays entering the object to output rays exiting the object. We are interested in computing the output ray without actually tracing the input ray through the object. This is achieved by adaptively sampling rays from multiple viewpoints in various directions, as a preprocessing phase, and then interpolating the collection of nearby samples to compute an approximate output ray for any input ray. In most cases, object boundaries and other discontinuities are handled by applying various heuristics. In cases where we cannot find sufficient evidence to interpolate, we perform ray tracing as a last resort. We provide performance studies to demonstrate the efficiency of this method.