2D cel and cartoon effects can easily be achieved with the VRayToonMtl. Use this material to make your scene get that hand-drawn look. Controlling the shadows and lights received by the material in combination with material transparency, gotten from Object or Material IDs, allows for fine-tuning the result. You can take advantage from the other standard V-Ray material options such as reflection, refraction, anisotropy, subsurface scattering and bump/normal mapping to set up the render to your liking.

Combine with VRayToon to add custom outlines for inky effect.

Diffuse Color – Base texture blended with the color from the diffuse ramp. See the Diffuse Color with Diffuse Ramp example below.

Amount – A multiplier for the base texture. 

Diffuse Ramp

The Diffuse Ramp controls the diffuse color based on the amount of light received. Position 0.0 maps to light intensity 0.0, position 1.0 maps to light intensity 1.0 and above. See the Diffuse Ramp example below.

Diffuse Ramp Offset – Grayscale texture used to offset the ramp position for the respective shaded point.

Diffuse output offset – Amount used to offset the ramp position by a static value. Useful when the ramp offset texture has values in [0, 1] range but the user wants values in [-0.5, 0.5] range instead.

Opacity Map – Assigns opacity to the material where white is completely opaque and black is completely transparent. You can also assign a map by clicking the check board button. This way you can create a material that has a non-uniform opacity.

Opacity Mode – Controls how the opacity map works.

Normal – The opacity map is evaluated as normal: the surface lighting is computed and the ray is continued for the transparent effect. The opacity texture is filtered as normal.
Clip – The surface is shaded as either fully opaque or fully transparent depending on the value of the opacity map (i.e. without any randomness). This mode also disables the filtering of the opacity texture. This is the fastest mode, but it might increase flickering when rendering animations. 
Stochastic – The surface is randomly shaded as either fully opaque or fully transparent so that on average it appears to be with the correct transparency. This mode reduces lighting calculations but might introduce some noise in areas where the opacity map has gray-scale values. The opacity texture is still filtered as normal.

Roughness Amount – Used to simulate rough surfaces or surfaces covered with dust (for example, skin, or the surface of the Moon).

Self-Illumination – The self-illumination color of the material. A texture map can be used for the self-illumination color by clicking on the check board box next to the color slider.

Self-Illumination GI – When enabled, the self-illumination affects global illumination rays and allows the surface to cast light on nearby objects. Note, however, that it may be more efficient to use area lights or VRayLightMtl material for this effect.

Compensate Exposure – When enabled, the intensity of the Self-Illumination will be adjusted to compensate the exposure correction from the VRayPhysicalCamera.

Shadow Blend On – When disabled, the shadow is considered as negative light amount and used by the diffuse ramp for color selection. When enabled, the shadow is rendered separately and can be blended with the diffuse color.

Shadow Color – Texture that modifies the shadow color.

Shadow Opacity – Strength of the shadow when blending with the diffuse color. A value of 0 means completely transparent shadow (or no shadow), while a value of 1 gives full shadow result. 

Replace light color – When enabled, the color for all lights illuminating an object with the current material is replaced by Light Color.

Light Color – Replacement color for the original light color.

Light Blend Mode – Mode for blending the original light with the diffuse color.

None
Normal 
Over 
Add 
Subtract 
Multiply 
Lighten
Darken
Saturate
Desaturate
Illuminate

Light Blend Intensity – Amount for blending the original light and the diffuse color. A value of 0 results in no light blending, while a value of 1 results in full light blending. A texture map can be used here. 

Get transparency ID from – Source for IDs used by the material transparency list. Currently supported sources are VRay Object ID and VRay Material ID.

This is a list of transparency amounts. A transparency value is used if an object/material with the respective value is behind the object with the current material.

Add New Item – Adds a new sub-texture to the Material Transparency list.

Used – Enables the sub-texture for rendering.

Value – For modes that use IDs, specifies the ID that will correspond to each texture.

Reflection Color – The reflection color dims the diffuse surface color.

Amount – A multiplier for the reflection color.

Specular Ramp

This ramp controls the specular color based on the amount of light received. Position 0.0 maps to light intensity 0.0, position 1.0 maps to light intensity 1.0 and above.

Reflection Glossiness – Controls the sharpness of reflections. A value of 1.0 means perfect mirror-like reflection; lower values produce blurry or glossy reflections.

Use Fresnel – When enabled, makes the reflection strength dependent on the viewing angle of the surface. Some materials in nature (glass, etc.) reflect light in this manner. Note that the Fresnel effect depends on the index of refraction as well.

Lock Fresnel IOR to Refraction IOR – Allows the user to unlock the Fresnel IOR parameter for finer control over the reflections.

Fresnel IOR – The IOR to use when calculating Fresnel reflections.

Highlight Type – Offers two modes to set the anisotropy highlight: 

Shaped highlight – Using a curve and points to set the highlights.
Textured highlight – Textured highlight allows a greater degree of control over the final look. It uses the same VRayMtl highlight anisotropy UV space to map a texture over the highlight.

Highlight Mask – When the Highlight Type is set to Textured, this parameter becomes active. Select a texture to use as a mask for the highlight.

Highlight Shape Preset – Presets for the curve that controls the shape of the highlight. The curve positions are mapped to points on a circle (the default highlight shape). When the Highlight Type is set to Textured, the presets of this option are ignored. 

Default 
Square 
Hexagon 
Heart 
Star 

Split U/Split V  – Amount for highlight splitting in the U/V tangent direction.

Highlight Rotation –  Amount for highlight rotation. A value of 0 means no highlight rotation and a value of 1 results in 360 degree rotation. 

Trace Reflections – Check in this option to enable reflections for the material.

Max depth – The number of times a ray can be reflected. Scenes with lots of reflective and refractive surfaces may require higher values to look correct.

Dim distance On – Enables the Dim distance parameter which allows you to stop tracing reflection rays after a certain distance.

Dim distance – Specifies a distance after which the reflection rays will not be traced.

Dim fall off – A fall off radius for the dim distance.

Reflect On Back Side – When disabled, V-Ray calculates reflections for the front side of objects only. Enable it to make V-Ray calculate the reflections for the back sides of objects too.

Affect Channels – Allows the user to specify which channels are going to be affected by the reflectivity of the material.

Color Only – The reflectivity affects only the RGB channel of the final render.
Color+alpha – Causes the material to transmit the alpha of the reflected objects, instead of displaying an opaque alpha.
All channels – All channels and render elements are affected by the reflectivity of the material.

Refraction Color –  Refraction color. Note that the actual refraction color depends on the reflection color as well.

Amount – This is the amount of the refraction color.

Refraction Glossiness – Controls the sharpness of refractions. A value of 1.0 means perfect glass-like refraction; lower values produce blurry or glossy refractions.

Refraction IOR – Index of refraction for the material, which describes the way light bends when crossing the material surface. A value of 1.0 means the light does not change direction. 

Fog Color – The attenuation of light as it passes through the material. This option helps simulate the fact that thick objects look less transparent than thin objects. Note that the effect of the fog color depends on the absolute size of the objects and is therefore scene-dependent.

Fog Multiplier – The strength of the fog effect. Smaller values reduce the effect of the fog, making the material more transparent. Larger values increase the fog effect, making the material more opaque.

Fog bias – Changes the way the fog color is applied. Negative values make the thin parts of the objects more transparent and the thicker parts more opaque and vice-versa (positive numbers make thinner parts more opaque and thicker parts more transparent).

Affect Shadows – This parameter causes the material to cast transparent shadows to create a simple caustic effect dependant on the refraction color and the fog color. 

On – Enables sub-surface scattering for the material.

Translucency Color – Normally the color of the sub-surface scattering effect depends on the Fog color; this parameter allows you to additionally tint the SSS effect.

Fwd/back coeff – Controls the direction of scattering for a ray. a value of 0.0 means a ray can only go forward (away from the surface, inside the object); 0.5 means that a ray has an equal chance of going forward or backward; 1.0 means a ray is scattered backward (towards the surface, to the outside of the object).

Scatter bounces – Controls how many times the rays bounce inside the object.

Scatter coefficient – The amount of scattering inside the object. 0.0 means rays are scattered in all directions; 1.0 means a ray cannot change its direction inside the sub-surface volume.

Thickness – Limits the rays that will be traced below the surface. This is useful if you do not want or don't need to trace the whole sub-surface volume.

Environment fog – When enabled, V-Ray traces direct lighting into the material.

Trace Refractions – Enables refractions for the current material.

Max depth – The number of times a ray can be refracted. Scenes with lots of refractive and reflective surfaces may require higher values to look correct.

Affect Channels – Allows the user to specify which channels are going to be affected by the transparency of the material.

Color Only – The transparency will affect only the RGB channel of the final render.
Color+alpha – This causes the material to transmit the alpha of the refracted objects, instead of displaying an opaque alpha. 
All channels – All channels and render elements are affected by the transparency of the material.

Dispersion – Enables the calculation of true light wavelength dispersion.

Dispersion Abbe – Allows the user to increase or decrease the dispersion effect. Lowering it widens the dispersion and vice versa. 

Map Type – Determines how the Map parameter is interpreted.

Map – Allows the user to select a texture for the bump or normal map. Leaving this unconnected turns off bump/normal mapping.

Bump Mult – A multiplier for the bump map effect.

Bump Shadows – When enabled, shadows are properly displayed, considering any bump surface they fall on. It's recommended to keep this parameter disabled.

Bump Delta Scale – This parameter can be decreased to sharpen the bump and increased to blur it.

Cutoff Threshold – A threshold below which reflections/refractions are not traced. V-Ray tries to estimate the contribution of reflections/refractions to the image, and if it is below this threshold, these effects are not computed. Do not set this to 0.0 as it may cause excessively long render times in some cases. This parameter is not available when the renderer is set to V-Ray GPU.

Double-sided – When enabled, V-Ray also shades the back-facing surfaces with this material. Otherwise, the lighting on the outer side of the material is always computed. This can be used to achieve a fake translucent effect for thin objects like paper.

Use Irradiance Map – When enabled, the irradiance map is used to approximate diffuse indirect illumination for the material. If this is off, Brute Force GI is used, in which case the quality of the Brute Force GI is determined by the Subdivs parameter of the Irradiance Map. You can use this for objects in the scene which have small details and are not approximated very well by the Irradiance Map.

Fix dark edges – When enabled, fixes dark edges that sometimes appear on objects with glossy materials.

Enabled – Enables the Toon Edges Override effect. See the Toon Edges Override example below.

Toon Edge Priority – The Toon Edge Priority is taken into consideration when two edges overlap. If the two objects come with the same priority, for each part of the edge overlapping the object, V-Ray looks into the edge's corresponding settings. However, if any of the objects has its priority set higher, the whole edge is drawn with its settings. All materials other than VRayToonMtl have their priority set to 0.

Line Color – The color of the outlines.

Line Width Multiplier – Scales the global toon effect line width multiplying it with a value from the range of 0 to 1. A value of 0 removes the outlines and a value of 1 renders the original line width specified in the VRayToon. You can connect a texture map here to control the line width.

Inner Line Control – Enable this option to get control over the inner edges.

Inner Line Color – The color of the inner edges' outlines. A texture map can be connected to this parameter.

Inner Line Width Multiplier – Scales the global width of the inner edges' outlines multiplying it with a value from the range of 0 to 1. A texture map can be connected to this parameter.

Outer Overlap Threshold – Determines when outlines are created for overlapping parts of one and the same object. Lower values reduce the outer overlapping lines, while higher values produce more overlap outer lines.

Normal Threshold – Determines at what point lines are created for parts of the same object with varying surface normals (for example, at the inside edges of a box). Lower values mean that only sharper normals generate an edge, while a value of 0.5 means that 90 degrees or larger angles generate internal lines. Higher values mean that smoother normals can also generate an edge. Don't set this value to pure 1.0 as this fills curved objects completely.

Overlap Threshold  – Determines when outlines are created for overlapping parts of one and the same object. Lower values reduce the internal overlapping lines, while higher values produce more overlap lines. Don't set this value to pure 1.0 as this fills curved objects completely.

Mask Color Threshold – Sets a threshold for the mask color. The range is between a value of 0, where there are no outlines created and 1 where there are many outlines created.

Mask Color – Uses a texture as a mask creating more outlines depending on the Mask Color Threshold value.