Matte Properties

V-Ray has its own matte system. You can control matte properties either on an object level through the Matte Properties node, or on a material level through the special V-Ray Mtl Wrapper.

Select the object you want to convert into a matte object and add this node to the Object Node Tree. Keep in mind that if the Matte Properties are not connected to a V-Ray Object Output node, they are not applied to the object.

UI Path: ||V-Ray Node Editor|| > Object > Add > Object Properties> V-Ray Matte Properties

Alpha contribution – Controls how the object appears in the alpha channel. Note that an object doesn't have to be a matte object for this setting to work - the alpha contribution can be changed regardless of whether the object is matte or not. A value of 1.0 means that the object appears normally in the alpha channel. E.g. if the object is opaque, the alpha is white; if the object is fully transparent, it does not influence the alpha channel at all. A value of 0.0 means that the object, regardless of its actual opacity, is not present in the alpha channel, although it is present, as usual, in the RGB channel. A value of -1.0 means that the object's alpha is cut out from the alpha of the objects behind it. For example, if the object is fully opaque, its alpha is black, and it blocks the alpha of the objects behind it, whereas if the object is fully transparent, it does not influence the alpha of the other objects at all. Note that adding matte properties to an object does not change its appearance in the alpha channel. You need to explicitly change its alpha contribution. 

Shadows – When enabled, the matte object receives shadows.

Affect alpha – When enabled, makes the shadows affect the alpha contribution of the matte surface. Areas in perfect shadow produce white alpha, while completely unoccluded areas produce black alpha. Note that GI shadows (from skylight) are also computed. However, GI shadows on matte objects are not supported by the Light Cache GI engine, when used as the primary engine. You can safely use Light Cache with matte surfaces as a secondary engine.

Shadow Tint Color – Specifies the color of the shadows.

Shadow Brightness – An optional brightness parameter for the shadows on the matte surface. A value of 0.0 makes the shadows completely invisible, while a value of 1.0 shows the full shadows.

Reflection Amount – If the material of the object is a reflective V-Ray material, this controls how much of the reflection is visible on the matte object.

Refraction Amount – If the material of the object is a refractive V-Ray material, this controls how much of the refraction is visible on the matte object.

GI Amount – Controls how much of the indirect illumination received by the object is visible on the matte.

No GI on Other Mattes – When enabled, the object appears as a matte object in reflections, refractions, GI, etc., for other matte objects. Note that if this is enabled, refractions for the matte object might not be calculated. (The object appears as a matte object to itself and is not able to "see" the refractions on the other side.)

Matte for Secondary Rays – Normally, the base material is used when an object with a Mtl Override is seen through reflections/refractions. Enable this option, if you want the Override Material to show the environment when seen through reflections/refractions. V-Ray can also do projection mapping to increase the realism.