When creating a matte object with VRayMtlWrapper and using the V-Ray CUDA engine, please note the following:
Alpha Contribution should be set to -1.0;
The Matte Surface, Affect Alpha and Shadows options should be enabled.
In V-Ray 5, matte objects always show the background. Since V-Ray 6, however, the matte objects show the V-Ray Dome light instead of the background when a V-Ray Dome light is present in the scene.