You use the Left Mouse Button (LMB) for selection and deselection of geometry (points and polygons, and Groups). Try the following:

Click the LMB near a point to select either the point (in Point-Edit Mode) or the group that contains the point (in Group-Edit Mode).
Hold down either Control-Key while clicking with the LMB to avoid selecting more than one polygon at a time.
Hold down the LMB while dragging the mouse to rubber-band a box around the points / polygons you wish to select.
Hold down the LMB and either Shift-Key while dragging the mouse to draw a freehand lasso around the points / polygons you wish to select.
Hold down either Alt-Key while performing any of the above operations to deselect geometry instead of selecting it.
Use the Camera / Box-Select Partial and Camera / Lasso-Select Partial commands to control whether polygons that are partially contained within rubber-band boxes or lassos are selected.